Optimal Crowd-Powered Rating and Filtering Algorithms

We focus on crowd-powered ltering, i.e., ltering a large set of items using humans. Filtering is one of the most commonly used building blocks in crowdsourcing applications and systems. While solutions for crowd-powered ltering exist, theymake a range of implicit assumptions and restrictions, ultimately rendering them not powerful enough for real-world applications. We describe two approaches to discard these implicit assumptions and restrictions: one, that carefully generalizes priorwork, leading to an optimal, but oŸentimes intractable solution, and another, that provides a novel way of reasoning about ltering strategies, leading to a sometimes suboptimal, but e›ciently computable solution (that is asymptotically close to optimal). We demonstrate that our techniques lead to signicant reductions in error of up to 30% for xed cost over prior work in a novel crowdsourcing application: peer evaluation in online courses.